Repository navigation
WO20: /benchmarks page — reproducible runtime benchmarks (scripts + script-generated data) - #239
Merged
Merged
Conversation
…rated Deliverable A of WO20 (issue #238): published, reproducible evidence for SOTA 2027. scripts/benchmarks/ — checked-in drivers per runtime plus one orchestrator: - corpus parse, median-of-3, the parity-README methodology (in-process for ruby/ts/py, iscrs parse wall for rs) - golden replay: the RS golden capture (isc-wf0.jsonl, 12,839 rows) through each runtime; rs via native iscrs capture - transliteration throughput on a committed fixed input set (fixtures/input-set.jsonl, every 25th golden row, frozen); rs has no batch CLI, so its throughput derives from the capture leg and is marked as such generate.mjs runs every leg strictly serially and writes src/data/benchmarks.json — numbers are never hand-edited; the page renders only what the script produced. The scripts read the four runtime trees and the map corpus from sibling checkouts (env-overridable INTERSCRIPT_* and RUBY_BIN); the requirement is documented in scripts/benchmarks/README.md and on the page itself. The page keeps the audit-day parity-README table (2026-11-13) frozen as the baseline and tracks current-HEAD above it, with per-tree commit provenance and a machine profile. Generated from fresh origin/main clones: rs 0.3.2 (74b4b8d), ts 5.10.0 (b74cfdaf), ruby 3.7.2 (4c47eb52), py 0.1.0 (5781a996, the unmerged py/ISC branch of interscript-ruby), maps c55bc5c, on an Apple M1 Max / ruby 3.4.8. Findings surfaced by the first run (recorded on the page): py diverges on 3 of 12,839 golden rows (the new Ў rows of gki-bel/mvd-bel/odni-bel); ruby origin/main fails to parse 120/428 corpus maps under ruby 4.0.7 (CI covers 3.3/3.4 — the benchmarks run on 3.4.8, which parses 428/428).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WO20 deliverable A:
/benchmarkspage + checked-in reproducible scripts (issue #238)What's here
scripts/benchmarks/— checked-in, runnable benchmark drivers per runtime plus one orchestrator:.isc, median-of-3, exactly the parity-README methodology (in-process timing for ruby/ts/py;iscrs parsewall clock for rs);isc-wf0.jsonl) through each runtime (in-process; rs via its native single-processiscrs capture);fixtures/input-set.jsonl, every 25th golden row, sampled once and frozen) so the series stays comparable as the golden evolves. rs has no batch transliteration CLI, so its throughput is derived from the capture leg and marked as such.generate.mjsruns every leg strictly serially (one runtime at a time — the parity methodology) and writessrc/data/benchmarks.json. Numbers are never hand-edited; the page renders only what the script produced./benchmarkspage — current-HEAD numbers, machine profile, per-tree commit provenance, the parity-README audit-day table (2026-11-13) kept frozen as the baseline, and exact reproduce commands.src/data/benchmarks.jsonis the output of the scripts on this machine, and CI (astro check + build) passes.Sibling-checkout requirement
The scripts read the four runtime engines and the map corpus from sibling checkouts of the monorepo layout (documented in
scripts/benchmarks/README.mdand on the page). All locations are env-overridable (INTERSCRIPT_MAPS/RS/TS/RUBY/PY,RUBY_BIN). First run used freshorigin/mainclones — the localinterscript-ts/interscript-ruby/interscript-rscheckouts sit on diverged branches, and the page records every tree's commit so provenance is explicit:wo17/py-override-placementbranch of interscript-ruby)Machine: Apple M1 Max, macOS, ruby 3.4.8 (CI-covered; see finding below), node 24, python 3.10.5.
Results (current HEAD, median of 3, serial)
vs the frozen audit-day baseline: rs 79 s → 26 s; py 81 s → 4.7 s (the generated-parser branch); ts 16 s → 15 s; ruby 161 s → 167 s.
Findings surfaced by the first run (not fixed here)
Ўrows of the three thin ALA-LC Belarusian maps (gki-bel,mvd-bel,odni-bel), the newest corpus additions (maps v3.2.3 exhaustive per-row tests); ts and ruby replay them byte-exact. The run's guard reports divergence rather than hiding it.interscript3.5.0 andinterscript-maps3.2.0 gems installed) the sweep fails 120/428 corpus maps withMapNotFoundErrorfrom parse-time dependency locate (31 still fail even with an explicitInterscript.load_path). On CI-covered ruby 3.4.8 — no interscript gems installed,ruby -I <tree>/lib— the identical sweep parses 428/428 and replays the golden byte-exact. The benchmark scripts therefore exposeRUBY_BINand record the binary in the machine profile; the root cause of the 4.0.7 dependency-locate behavior is flagged for the owner, not fixed here.interscript-pymain still carries the legacy.impengine and cannot run the ISC corpus at all.Verification
npx astro check— 0 errors;npm run build— complete (benchmarks page prerendered);npm test— 239 passed (the one failing file,test/api-endpoints.test.ts, fails identically on a cleanmaincheckout — pre-existing local-env server-start flake; CI does not run vitest).npm run format:check/lint— my files clean; the repo-wide drift (25 files / 1 error) pre-exists onmainand is untouched.